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Helping beginning qualitative researchers critically appraise qualitative 
research articles is a common learning objective for introductory 
methodology courses. To aid students in achieving competency in 
appraising the quality of qualitative research articles, a multi-part activity 
incorporating the Critical Appraisal Skills Programme’s (CASP) Making 
Sense of Evidence Tool: 10 Questions to Help You Make Sense of 
Qualitative Research to evaluate the articles is shared. A Contextualized 
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Introduction and Rationale 

Helping beginning qualitative researchers critically appraise qualitative research 
articles is a common learning objective for introductory methodology courses. The 
learning activity allows students to become acquainted with the fonn of qualitative 
research reports, to identify key parts of a qualitative research article, and to judge the 
quality of authors’ representations of methods and findings. The developmental levels of 
beginners may also make the activity a challenge due to the students’ lack of fundamental 
knowledge of what elements constitute a qualitative research article let alone what may 
define a “best practice” in qualitative research reports. In addition, the shear variety of 
qualitative research methods and criteria for assessing quality of qualitative research 
result presentations (Cohen & Crabtree, 2008) can make acquisition of competencies in 
this area quite challenging. 

To aid students in achieving competency I have designed and used an assignment 
called “Appraising the Quality of Qualitative Research Articles” to evaluate qualitative 
research reports. The assignment consists of multi-part task in which students locate three 
qualitative research articles, assess the retrieved articles using the Critical Appraisal 
Skills Programme’s (CASP) Making Sense of Evidence Tool: 10 Questions to Help You 
Make Sense of Qualitative Research 

( http://www.phru.nhs.uk/Doc Links/Qualitative%20Appraisal%20Tool.pdf) , write of 
their assessment of each article, and compare and contrast the relative quality of the 
articles. 
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In this paper I will share this assignment as a reusable learning object—“self¬ 
describing, self-contained small chu nk s of learning that accomplish a specific learning 
objective (Oakes, 2002) or as Wiley (2002a, p. 6) describes them, “any digital resource 
that can be reused to support learning” (Chenail, 2004, p. 113). The style in which I will 
present this learning object is known as CLOCKS or “Contextualized Learning Objects 
for Constructing Knowledge.” In contrast to other ways of rendering learning objects, 
CLOCKs are described in tenns of their contexts, evaluation components, exemplary 
outcomes, and options for customizing the parts of the CLOCK (Chenail, 2004). 

Appraising Articles with the CASP Tool 

I drew upon the Critical Appraisal Skills Programme’s (CASP) Making Sense of 
Evidence Tool: 10 Questions to Help You Make Sense of Qualitative Research to give the 
appraising assignment structure and to help students engage qualitative research articles 
efficiently. I have also found the use of the CASP tool provides students with the 
scaffolding they need to begin to make distinctions within qualitative research articles 
and to have a system within which they can compare and contrast articles relative quality. 
The CASP tool contains 10 questions designed to help readers appraise qualitative 
research reports broadly in terms of rigor, credibility, and relevance. The tool is not 
meant to be a definitive guide, but rather presents probes students can use to identify the 
salient features of a qualitative research article and to note what elements may be missing 
from a report. 

The CASP tool’s ten questions help the students begin to identify the basic parts 
of a research article (e.g., sample, data collection, data analysis, and results) and to 
consider how well the author presented the steps taken in each of these areas during the 
study and the rationales for making each of these decisions. As the students query the 
article from the perspective of the CASP’s questions, they can begin to appreciate the 
quality of these texts and to see how the theoretical and conceptual material presented in 
an introductory qualitative research textbook is operationalized (e.g., Creswell, 2006) and 
practiced in actual qualitative research articles. 

The use of the CASP tool also helps the assignment become a mini-data analysis 
activity as students must first conduct a within-case analysis of each individual article 
and then an across-case analysis in which they compare and contrast the three articles. In 
the within-case analysis the students use the ten CASP questions to analyze the contents 
of each article. For example, one CASP question asks the students to reflect upon the 
author’s attempt to present how ethical issues were managed in the research and another 
CASP question prompts the students to consider whether or not the author described a 
rigorous data analysis process. For each question the students render an evidence-based 
decision on the quality of the author’s presentation of the study. 

As they are rendering their findings, the students have to consider the overall 
quality across the ten CASP questions. This new focus helps them to shift their analysis 
from the ten questions which serve as categories to a more thematic analysis driven by 
the question of quality. The CASP tool suggests the students consider three overall 
quality measures to make this detennination: rigor, credibility, and relevance. For rigor 
the CASP tool asks the students to consider the thoroughness of the author’s accounts. 
For credibility, the students are asked to contemplate if the author presented the findings 
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in a meaningful fashion. For relevance, the students are encouraged to consider how the 
author established the usefulness of the results. These three factors can be used by the 
students as sub-themes of the overall detennination of an article’s quality. 

The students’ findings regarding the ten CASP categories and the three sub¬ 
themes for each article give the students points of relationship along which they can then 
judge the three articles collectively. They can compare and contrast the articles not only 
in terms of authors’ discussions of ethical concerns and data analysis rigor, but also along 
the lines of credibility and relevance. The rigor of the CASP-driven analysis helps the 
students to go beyond the simple repeating of the content of the articles and presenting an 
overall impression; and instead, deliver a data-driven account based upon a rigorous 
analytical process. 


Assignment Presentation 

In the qualitative research courses I teach I organize the syllabus around the 
program’s overall learning goals and objectives. I do this because I want to help the 
students to stay focused on the big picture in their degree or certificate program and to 
see how mastering the learning outcomes in this particular course will help them take the 
next step towards their overall goals (Chenail, in press). For example, we use the 
Appraising the Quality of Qualitative Research Articles in the Nova Southeastern 
University’s Graduate Certificate in Qualitative Research. The Certificate’s three goals 
for its students are to (a) gain insight into the philosophical and theoretical foundations of 
qualitative research; (b) distinguish and critique qualitative research approaches and 
products; and (c) apply best practices in qualitative research to design, propose, conduct, 
and compose qualitative research of their own. After reminding the student of the 
programmatic goals, I then connect these global goals with the course’s learning 
objectives. In this case, I show the students by accomplishing the learning objectives for 
the appraising assignment successfully they will be making progress towards the program 
goal of distinguishing and critiquing qualitative research approaches and products. I make 
this connection overt by using the following table in the syllabus to present the relevant 
Certificate goal, the appraising assignment’s learning outcomes, and the process by 
which the students’ competencies will be measures during the activity. 


Certificate Goal: Distinguish and Critique Qualitative Research Approaches and Products 
Assignment: Appraising the Quality of Qualitative Research Articles 

Learning Outcomes 

Direct Assessment Activity 

Assessment Method 

Locate and retrieve qualitative 
research articles. 

Compare and contrast defining 
attributes of selected qualitative 
research and evaluation 
methodologies. 

Describe methodology-specific 
techniques for sampling, data 

Students will select three 
qualitative research papers 
which present results from 
qualitative data analysis, 
analyze the three papers 
using the CASP tool, and 
compare and contrast the 
quality of the three papers 
based upon the results of 
their CASP tool analysis in a 

A criterion-based 
rubric is used to assess 
students’ abilities to 
compose a 12 to 15 
page paper in 
compliance with APA 
guidelines in which 
they will articulate their 
methodology for 
finding candidate 
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generation, collection, and 

12 to 15 page paper. 

papers, their rationale 

preparation, data analysis, and 


for selecting their 

interpretation and representation. 


papers, the quality of 
their analysis, the 

Conduct within-case and across case 


quality of their 

document analyses to determine 


comparative analysis, 

relative quality of qualitative research 


and the coherence of 

articles. 


their discussion about 
the findings of their 

Report the findings of their analysis. 


analysis. 


After situating the assignment within the Certificate and Course learning 
outcomes, I then present the activity as follows: 

Appraising the Quality of Qualitative Research Articles - xx points 

As a qualitative researcher, it will be critical for you to collect, read, and assess 
published reports including ones employing qualitative research methodologies. Finding 
such papers can help you (a) identify relevant evidence for your research studies, (b) 
appreciate the utility of qualitative research methodology when it comes to studying 
various education topics, and (c) select an appropriate methodology to address your own 
research questions. 

When you locate pertinent research papers, it is critical that you can assess the 
quality of these published accounts and synthesize your understandings of these sources. 
This objective can be even more challenging in qualitative research given the variety of 
methodologies, styles, and philosophical approaches, as well as the uneven nature of the 
quality of some published results (Barroso, Gollop, Sandelowski, Meynell, Pearce, & 
Collins, 2003; Flemming & Briggs, 2007; McKibbon, Wilczynski, & Haynes, 2006). To 
help you to develop a critical eye for evaluating qualitative research you will be asked to 
appraise the individual and comparative quality of three qualitative research papers you 
have collected. To help you in discerning the quality of your chosen papers, you will use 
the Critical Appraisal Skills Programme’s (CASP) Making Sense of Evidence Tool: 10 
Questions to Help You Make Sense of Qualitative Research 
(http://www.phru.nhs.uk/Doc Links/Quahtative%20Ar>praisal%20Tool.r>df) to evaluate 
the articles. 

The assignment also serves as an introduction to the conduct of within-case and 
across case qualitative data analysis of documents and the reporting of your resultant 
findings. In the case of this assignment, your CASP within-in case analysis of each 
qualitative research paper will help you to open up the article, separate its content into 
meaningful categories, render your findings for each of these categories, and then declare 
your pronouncement of the article’s quality by identifying patterns across the CASP- 
determined categories. Finally, after you have offered your findings for each article, you 
will then conduct an across-case analysis of the three articles and discuss the comparative 
quality of your three qualitative research articles. 
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To successfully complete this assignment you must 

1. Select three qualitative research papers which present results from qualitative 
research methodology relevant to nursing education. 

2. Analyze the papers using the CASP tool. 

3. Discuss the papers’ within-case and across case quality based upon the results of 
your CASP tool analysis in a 15 page paper. 

4. Compose your paper in compliance with the American Psychological Association 
(2001) Publication Manual of the American Psychological Association (5 th ed.). 

Your 15 page paper must include the following elements: 

1. Title Page 

2. Abstract 

3. Introduction to the three articles 

4. Search plan for locating and rationale for selecting your papers 

5. Methodology for conducting the within-case and across case analysis including 
your quality control procedures 

6. Results of the CASP analysis of your three papers 

7. Findings and discussion of your three papers’ comparative quality 

8. References 

9. Appendix 

a. Completed CASP Tool analysis of your papers 

b. Copy of your papers 

The 15 pages do not include your title page, abstract, references, or appendix. 

Your work on the Critique and Comparison Paper assignment will be graded on 

1. Title page - x points 

2. Abstract - x points 

3. Introduction - x points 

4. Search plan for finding and rationale for selecting your papers - x points 

5. Methodology for conducting your analyses - x points 

6. The quality of your within-case CASP analysis (including the Appendix) - x 
points 

7. The quality and coherence of your across-case comparative quality discussion - x 
points 

8. The compliance of your references with APA conventions - x points 


The following rubric will be used to assess your progress on this assignment: 
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Appraising the Quality of Qualitative Research Articles 

Total: xx out of xx Points 

Section Goals 

Strengths 

Changes 

Points 

Title Page (x points): Clearly 
identify the subject of your paper 
(e.g., Appraising Assignment). 
You can also add information 
describing the content of the 
three papers being compared 
(e.g., Comparing the 
Methodological Quality of Three 
Qualitative Research Domestic 
Violence Studies). Complies 
with the directions outlined in the 
Publication Manual of the 
American Psychological 
Association (APA, 2001) 

Sections 1.06 (pp. 10-12), 5.03 
(p. 286), 5.04 (pp. 286-287), 5.06 
(p. 288), and 5.15 (pp. 296-298) 
(i.e., includes a header with page 
number, a running head, the title 
of the paper, the author’s name, 
and the author’s affiliation). 




Abstract (x points): In 120 

words or less present the key 
points you will make in the 
major sections of the paper; 
therefore you can write one 
sentence each summarizing the 
main points of the Introduction, 
the Rationale, the Comparative 
Analysis, and the Discussion. 
Complies with the directions 
outlined in the Publication 

Manual of the American 
Psychological Association (APA, 
2001) Sections 1.07 (pp. 12-15) 
and 5.16 (p. 298). 




Introduction (x points): 

Introduce the reader to the 
assignment and the three papers 
selected and tell the readers what 
you plan to tell them in the rest 
of the paper. 
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Methodology for locating 
relevant qualitative research 
papers (x points): Explain the 
procedures you followed that 
allowed you to locate these 
papers (e.g., describe your search 
terms, databases accessed, or 
other search strategies employed 
to narrow or re-focus your 
search). 




Rationale for selecting the 
three qualitative research 
papers (x points): Explain why 
you selected the papers you did 
(e.g., describe your inclusion 
criteria and explain how these 
papers met them). 




Comparative analysis of the 
three qualitative research 
papers (x points): Using the 
results of the CASP analysis of 
the three papers you selected, 
compare and contrast the quality 
of the three papers from a 
methodological perspective. 
Support your findings by using 
quotations and page citations 
from the papers and the CASP 
tool and any other relevant 
resources. 




Discussion of the comparative 
quality of the three qualitative 
research or program 
evaluation papers (x points): 

Tell the readers what you have 
told them and come to a 
conclusion as to the comparative 
methodological quality of the 
three papers: Was one paper 
better methodologically than the 
others, were all papers of equal 
methodological quality, or were 
all papers of poor 
methodological quality? Also 
explain what you learned about 
quality in qualitative research 
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papers from reading and 
appraising these papers including 
any limitations to your analysis 
and what questions still remain 
concerning appraising qualitative 
research articles. 




References (x points): Include 
all references cited in the paper 
including the appendices. This 
means a citation for the CASP 
should also appear in your 
Reference section. Complies 
with the directions outlined in the 
Publication Manual of the 
American Psychological 
Association (APA, 2001) 

Sections 1.13 (p. 28) and 5.18 (p. 
299) and Chapter Four (pp. 215- 
281). 




Appendices (x points): Include 
a copy of all three papers and 
their individual CASP analyses. 

In your CASP analyses, support 
your findings by using quotations 
and page citations. Complies 
with the directions outlined in the 
Publication Manual of the 
American Psychological 
Association (APA, 2001) 

Sections 1.14 (pp. 28-29) and 

5.19 (pp. 299-300). 





Assessment 

In presenting this assignment I think it is important to share the criterion-based 
rubric with the students so they can see how their work will be assessed. Students can 
also be encouraged to use the rubric as a self-assessment tool for them as they write and 
revise their work. 

I use a simple rating fonn rubric (Middle States Commission on Higher 
Education, 2007) because it allows me to provide specific feedback to the students. I try 
to note strengths for each section and identify ways in which the element can also be 
improved. The specificity of the prescriptions I share in the “Changes” column helps the 
students focus on problems in their papers and provides guidance for the students to 
revise their submissions. I also assign a point total for each section and an overall point 
total for the activity. The mixture of the strength and changes comments with the point 
total helps to give students both a qualitative and a quantitative assessment of their work. 
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When a student turns in their assignment electronically I review the paper using 
various Microsoft® Office editing tools such as “track changes,” “insert comments,” and 
“highlight text” to embed my comments and suggestions in the text as well as in the 
rubric. Students receive a summary of their performances, suggested changes, and a 
quantitative score. In the cover email the students are given the option to revise and 
resubmit their assignments until they can clearly demonstrate that they had mastered the 
competencies and had earned the points they wanted to accrue for that part of the 
assignment. 

In assessing each component of the assignment I look for the prescribed elements 
as expressed in the directions and reinforced in the rubric. Maximum points are awarded 
when students (a) address all required elements; (b) support their assertions with excerpts 
from the articles and/or their analyses; (c) provide citations from third-party sources (e.g., 
course textbook) to support their observations; (d) write clear and coherent sentences, 
paragraphs, and sections; and (e) compose their work in compliance to the APA style 
manual. 

When reviewing the students’ CASP analysis for each article I play close 
attention to how the students answer each CASP question to observe if the student 
renders a clear finding and supports this pronouncement with evidence from the article 
itself. Students can demonstrate evidence of this competency by showing marked up 
articles with the material addressing each CASP noted and CASP worksheets where they 
record their determinations to each question with supporting quotes and page numbers 
from the articles. I try not to be overly prescriptive in guiding the students when they 
conduct their analyses in order to learn how they go about carrying out their coding and 
analysis. In doing so, each time I use this exercise I usually learn some novel way of 
analyzing textual data due to the students’ creativity. 

I use the same openness to execution when it comes to how the students render 
their appraisal of the three papers’ relative quality. I only ask that the students make a 
pronouncement of quality and provide evidence to support their findings. Some students 
have employed a mixed-method approach by assigning a numeric score for each CASP 
tool question and ranking the quality of the papers quantitatively and articulating the 
rationale for the scores qualitatively by sharing exemplars from the articles and 
commenting on their qualitative differences. Other students have used more global 
assessments of the articles’ quality by providing narratives of the articles’ strength and 
weaknesses along with supporting quotes. In both examples, the students successfully 
applied criteria to transform the articles’ content into categories, themes, and/or 
composite scores allowing them to compare and contrast the material and ultimately 
appraise the articles’ quality. 

In the Discussion section I ask the students explore the limitations of their 
findings and share what questions about the process still remain for them. This practice 
helps to prepare them for future qualitative research studies wherein they will have to 
address these two areas again. 
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Pedagogical Suggestions 

Each time I assign this exercise I leam some new ways to help make the activities 
work better for the students and myself. Here are some of the pedagogical steps I have 
used to improve the assignment and to address some difficulties students and I have had. 

1. Students lacking database searching experience may need to be shown how to 
access and utilize basic search functions. I have used Google Scholar 
(http://scholar.google.com ) in class to show students the basics of searching and 
retrieving articles online. 

2. Students can be asked to collect articles with no specific inclusion criteria other 
than the paper reports findings produced via the application of a qualitative research 
methodology. They can also be asked to collect articles in which the researchers used 
similar methodologies or studied similar phenomenon. 

3. Because the CASP tool is contained in a Adobe Acrobat “PDF” file, students 
may need to create a Microsoft Word version of the CASP tool if they wish to type in 
their findings; otherwise, they can print off the CASP tool and write in their results. 

4. When assessing the individual articles, students with a “full version” of Adobe 
Acrobat can use Acrobat’s editing features to insert their notes into the files themselves. 

5. The structure of assignment lends itself to a multiple submission/feedback 
process so students can turn in parts throughout the length of the course. The following is 
one suggested way of dividing up the assignment: (a) Student turns in candidate articles 
along with draft APA formatted references; (b) Student turns in one CASP analysis at a 
time to show the analysis and the draft write up of the results; and (c) Student turns in 
draft of the comparative article analysis. For each submission, the rubric can be used to 
present preliminary assessments of the work. As students revise and resubmit their parts, 
the “track changes” feature of Microsoft Word can be used to create an audit trail for the 
assignment. 

6. As an optional component students can keep a journal in which they describe 
the procedural steps they took to complete the assignment along with personal reflections 
they had on conducting the project. 

7. The assignment pairs well with a good introduction to qualitative research text 
which can provide students with background infonnation on basic qualitative research 
methods and procedures. The introductory material can also include writings on how to 
read a qualitative research article (e.g., Greenhalgh & Taylor, 1997) and a “read-along” 
in which the students and professor read through an article together and make note of the 
main features of the paper and offer assessments of the articles strengths and weaknesses. 
Students can cite these foundational sources to support their findings of quality in the 
articles. For example, the second question in the CASP tool directs the student to 
examine how the author described the appropriateness of the qualitative methodology 
used in the study. Once this infonnation is located the student can compare the rationale 
given in the article with rationales shared in a basic qualitative research textbook (e.g., 
Creswell, 2006). When writing up the assessment of the article, the student can present 
the basic prescription the book’s author shared when to use a qualitative research 
methodology, describe the article’s author depiction of the rationale, share a quote from 
the article evidencing this description, and share a pronouncement of the author’s 
perfonnance within the context of the CASP tool and the reference book. 
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Discussion 

I have used variations of this assignment over the past four years with master’s 
and doctoral students during their first qualitative research courses. In earlier iterations I 
had students retrieve and analyze five articles, but I have now found three articles suffice 
in giving students enough detailed exposure to qualitative research reporting variety and 
enough candidates to assess and compare for them to demonstrate competencies for these 
learning objectives. In some introductory courses the students also had to conceive and 
compose a qualitative research proposal. In those courses I have suggested students use 
papers they have collected for their proposals’ literature reviews in the appraisal 
assignment too. 

Students who struggle with this assignment can be divided into two general 
groups. Some students seem to have difficulties using different technological means to 
accomplish different parts of the exercise. For example some students appear to have 
little experience searching for materials in databases, or downloading papers from an 
electronic library, or using electronic means to analyze or edits documents. For this 
group, extra coaching provided by me or our technology and library help desk staffs 
along with supplemental guiding materials can assist in mastering these supporting 
competencies. 

The other group consists of students who struggle with reading critically and 
writing clearly. I group these two challenges together because it usually seems to hold 
that students who have trouble reading others’ work with a critical eye appear to lack that 
same skill when it comes to reading and writing their own texts. In the case of this second 
group I find breaking down the assignment into smaller pieces works well. For example, I 
will ask the student just to read the article to locate material that appears to respond to the 
first CASP tool question and to render an opinion to the article’s performance in this one 
category. The student would then turn in their analysis and finding for this question so I 
can provide feedback and guidance one area at a time until they can show they can 
complete the rest of the activity with less oversight. For the writing portion, I encourage 
the students to submit paragraphs, sections, and other smaller portions of the complete 
paper so adjustments can be made earlier than later. 

This assignment is still a work in progress for me. The CASP instrument makes 
for a fine training tool, but it can also skew students’ perceptions of what stands for 
quality in qualitative research. I constantly remind students of this deficit and the CASP 
tool itself expresses this caveat, but the exercise can also leave students with an overly 
narrow view of what qualitative research is and is not. There is also the debate of whether 
or not students should be reading less than exemplary examples of qualitative research. 
Reading such articles can help students to learn the weaknesses as well as the strengths 
and hopefully, they will remember to embrace the strengths in their own work. But as one 
student taught me last summer, strengths and weaknesses can be relative. 

In our PhD in nursing program one doctoral student selected three nursing 
education research papers that spanned a decade. She quickly noted in her report that the 
newer paper was in greater compliance with the CASP tool questions than its older 
counterparts. This observation helped her to organize how she presented the three articles 
in her paper and how she detennined the relative quality of the three publications. By 
drawing this temporal distinction she was able to gain an historical perspective on the 
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development of qualitative research report writing practices as well as on the evolution of 
the criteria with which we use to evaluate these articles. In doing so, she was able to gain 
an appreciation for the earlier works by situating them in a particular time and place even 
though by contemporary prescriptions and practices they now seemed lacking in some 
fashion. In such a way she was able to gain an appreciation for what those authors did 
and did not do in their reports. Because of this thoughtful appraisal her findings helped to 
remind me of the importance of context in conducting qualitative data analysis and that 
quality is something about which we have to remember its meaning in context all of the 
time. 
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